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Abstract 

Association Rules are a basic concept of data mining. They are, however, 
not understood as logical objects which can be used for reasoning. The purpose 
of this paper is to investigate a model based semantic for implications with cer- 
tain constraints on their support and confidence in relational data, which then 
resemble association rules, and to present a possibility to decide entailment for 
them. 



1 Introduction 

Association Rules are a basic pattern in data mining which has undergone extensive 
research during the last decades. Thereby a major focus has been on the question how 
to represent the set of all association rules as concisely as possible, as this set might be 
too large to be practically helpful. However, in those considerations representations 
based on model semantics have not been used. Instead, rules based formalisms for 
infering association rules from others have been developed [U[S] . One notable exception 
is the work of Luxenburger [5] who investigated partial implication, which are closely 
related to association rules and to the constrained implications we shall introduce in 
this paper. 

In this work we want to investigate this model based approach. For this, we shall 
develop a straight forward model based semantic for constrained implications, which 
shall replace the original notion of association rules. The main focus of our work then 
shall be the investigation on whether a set of constrained implications entails another 
constrained implication or not. We shall show that this question can be rephrased in 
terms of two linear programs, which then can be solve in polynomial space. 

We shall start our investigation with a short introduction into Formal Concept 
Analysis pQ , which we are going to use as the basic formalism to formulate our consid- 
erations. After this, we introduce the classical concepts of support and confidence. We 
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then investigate constrained implications and formulate a semantic based on models 
for constrained implications. We shall finally formulate the entailment problem for 
constrained implications. 

The next step shall then be an equivalent reformulation of a certain instance of 
the entailment problem into a set of two linear programs. The reformulation we are 
going to make is quite natural, however the resulting pair of linear programs may be 
exponentially large in the size of the original input. Nevertheless, as we shall show 
subsequently we are able to show that we can solve both linear programs in space 
polynomial in the size of the input. Hence, entailment of constrained implications is 
decidable in PSpace. 

2 Implications with Support and Confidence 

Let G and M be two sets and let i" C G x M. We shall call the triple K = (G, M, I) a 
(formal) context and shall associate with it the following interpretation: The elements 
g G G are the objects of K, the elements m G M are the attributes of K and we shall 
say that g has the attributes m if and only if (g,m) G I. In that case we may also 
write g I m. The set of objects of a formal context K may be denoted Gk, likewise 
Mk denotes the set of attributes of K and Ik denotes the incidence relation of K. 
Let us fix a formal context K and let A C Mk. We shall denote with 

A' :={geG\\/meA:gIm} 

the set of all common objects of all attributes of A. Likewise, for a set B C G(K) the 
set of common attributes of all objects in B is denoted by 

B' := { m G M \ Mg G B : g I m }. 

The sets A' and B' are called the derivations of A and B in K, respectively, and 
accordingly the operators •' are the called derivation operators of IK. 

Let A, B C Mr. We shall call the pair (A, B) an implication of K and denote it 
with A — > B. The set of all implications of K is denoted by Imp(K). We shall say 
that the implication A — > B holds in K, denoted K \= A — > B, if and only if every 
object g G Gk that has all the attributes of A also has all the attributes of B. Using 
the derivation operators, we can comprehensively write this as 

K\=A — > B <^=> A' C B' . 

Now let K be a finite and non-empty formal context, i. e. both Gk and Mk are now 
assumed to be finite and non-empty sets. The support of A is defined to by 



supp K (A) := 
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Similarly, the support of the implication A — > B is set to be 

supp K (A — > B) := supp K (A). 

Furthermore, the confidence of the implication A — > B is given by 

co „ W ^ B);= MUB)1 

if \A'\ =t 0. Otherwise, conf K (A — > B) := 1. 

We are now ready to introduce the notion of constrained implications. 

Definition 1 Let 1 be a finite and non-empty formal context. Then a constrained 
implication of K is a triple (A — > B, s, c) where A — > B 6 Imp(K) and s,c£ [0, l]nQ. 

A constrained implication r = (A — > B, s, c) is said to have minimal support 
so and minimal confidence cq if s > sq and c > cq. We shall denote the set of all 
constrained implications of K with minimal support s and minimal confidence Co by 
conImp sc (K). 

Constrained implications resemble the notion of association rules, but in a more 
formal setting. In particular, we can ask whether a constrained implication holds in 
an arbitrary formal context or not. 

Definition 2 Let K be a finite and non-empty formal context and let r = (A — > 
B, s, c) be constrained implication of K. Then r holds in IK, written as IK |= r, if and 
only if 

supp K (A — > B) > s, 

conf K (A — > B) > c. 

Let us fix a set C = { (Ai — > B i7 Si,Ci) \ i 6 / } of constrainted implications and 
let r = (A — > B,s,c) be another constrainted implication. The problem we want to 
investigate in the sequel is the following. 

Definition 3 (Entailment for Constrainted Implications) 

Given C and (A — > B,s,c), does then for every formal context K hold that 

K\= £ K\= (A — > B, s, c), 

i. e. does £ entail (A — > B, s, c)l 

We shall write £ \= (A — > B,s,c) if and only if £ entails (A — > B,s,c). An instance 
of the entailment problem is just a pair (£, r) of a set of constrained implications £ 
and a single constrained implication r. 



3 Linear Programs for the Entailment Problem 

The purpose of this section is to association with the set £ and the rule r = (A — > 
B,s,c) a pair of linear programs such that entailment can be decided by solving the 
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programs. Before we do so, however, we shall somehow restrict the sets of possible 
formal contexts K that we have to investigate to decide entailment. 

For this we observe that we can restrict the set of attributes of the formal con- 
texts occurring in the entailment problem to those attributes which actually occur 
somewhere in the implications involved. Define 

M := (AU B)uM AiU B t . 
Then the following statement holds: 

Lemma 4 Let £ and r = (A — > B,s,c) as before. Then £ \= (A — > B,s,c) if and 
only if for all formal contexts K with attribute set M it holds that 

K h £ => K h (A — > B, s, c). 



Proof The direction (=>) is clear. The main idea for the other direction is that impli- 
cations can only "restrict" the occurrence of attributes they contain. More formally, 
let us assume that £ ty= r. We shall show that then there exists a finite and non-empty 
formal context K with attributes set M such that K |= £ and K ty= r. Since £ \/= r 
there exists a formal context K with M C Mk such that K \= C and K ^ r. Let 
K be the formal context that arises from K by deleting all attributes Mk \ M. More 
formally, 

K := (Gk ,M,/k n (G Ko x M)). 
Now if X C M, then the derivation of X in K.q and K is the same, since 

g I Ko m g I K m 

holds for all g G Gk and m e X. Therefore, the support and confidence of all elements 
in C and of the rule r is the same on both K and K and hence K \= £,K y= r as 
required. □ 

We are now going to construct the aforementioned linear programs for our instance 
(£, r) of the entailment problem. 

Let K be a formal context with attribute set M. For every set A C M let xa be 
the number of objects g in K such that g' = A, divided by |G|, i. e. 

T ._ \{geG\ g ' = A}\ 
\G\ 

Now if Ti = (Ai — > Bi,Si 7 Ci) G £ is such that K |= r^, then 

supp K (Ai — > Bi) = supp K (Aj) > si, 
conf K (,4 i — > B t )>Ci. 
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This can we rewritten with the help of the variables x A as 



^2 X A > Si, 

ADA, 

x A - c t x A > o. 



(i) 



ADAiUl 



ADAi 



The second inequality stems from the fact that supp K ( J 4 i U Bi) — a supp K (Ai) > 0. 

Additionally, the variables x A satisfy the equation ^2 AC m x a = 1) which can be 
rewritten as two inequalities: 

^2 x A >l, 

ACM , N 

x . (2) 

ACM 

Finally all values x A satisfy x A > and x A 6 Q. 

Those inequalities can be rewritten in a compact form using matrices. Let A e 
q(2|/|+2)x2 |jj| an( j ^ g q2|/|+2 guc j 1 t ^at aforementioned inequalities are repre- 
sented as 

Ax > b, x > 0. (3) 

Thereby the vector x = (xi) i=1 2 i M shall be such that the variable Xi corresponds to 
the variable x A where the binary representation of i corresponds to the characteristic 
vector of A as a subsets of M, for some arbitrary but fixed linear ordering of M. 

Example 5 Let us illustrate the construction by means of a simple example. Let 

r={({a}-^{6},i/2 ) i/3),({a}-^{ C },i/3 ) V4)}. 
Then M = { a, b, c }. Let us order M by virtue of a > b > c. Then 



A = 



(o 
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1 


1 


1 \ 
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-1/3 


1 - 1/3 
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1 


1 





-1/4 





-1/4 





1 - 1/4 





1 - 1/4 
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-1 


-1 


-1 


-1 


-1 


-1 


- 1 / 



and 



b = 



o 

1/3 



1 

v-v 
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Here we have used the aforementioned convention, namely that the index i corresponds 
to the subsets of M whose characteristic vector with respect to the chosen linear 
ordering of M is the binary representation of i. Hence, i = corresponds to the empty 
set, i = 1 to the set { c } and so on. (} 

Theorem 6 Let £ — { (A4 — > Bi, Sj, Cj) | i G / } be a set of constrainted implications 
and let M := lJ igJ A{ U Bi. Then every context K with attribute set M and K |= £ 
induces a solution xk of {3J). Conversely, every solution x of (0) induces a nonempty 
collection K, of formal context K with attribute set M, IK |= £ and xk = x. 

We therefore see that there exists a one-to-one correspondence between non-empty 
classes of models of £ and the solutions of ([3]). 

Proof From the previous considerations we have seen that every formal context K with 
K \= £ gives rise to a solution of (J3J. 

For the converse direction let x be a solution of ([3]). Then the entries Xi of x 
are rational numbers. Let n be the least common denominator of all entries of Xi. 
Then define the formal context K x with attribute set M as follows: For every index 
i G { 1, . . . , 2' M I } we add Xi-n objects to K x whose intent is the set A that corresponds 
to the binary representation of i. Then |GkJ = n and we claim that K x is a model of 
£. 

For this let i G /. Then 

supp Kx (A; — > Bi) = ^ x j ^ s * 

j\A 3 DA t 

where Aj is the corresponding set for the index j. Likewise it follows confK x (A; — > 
Bi) > Ci, because x is a solution for (J3J). From the construction of K x and the 
construction of xjj x it is apparent that x = xjj x . □ 

The crucial observation now is that we can transform the problem of deciding 
entailment into a pair of linear optimization problems. The key idea is that we can 
read the entailment problem C \= (A — > B, s, c) as the question whether the minimal 
possible support for A — > B in every model of £ can be lower then s; similarly, one 
can ask what the minimal confidence of A — > B in all models of £ is lower than c. 
Therefore, entailment is equivalent to solving two linear optimization problems over 

<Q>. 

Theorem 7 Let £ U { (A — > B,s,c)} be a set of constrainted implications, \M\ = 
[J ieI Ai U Bi U A U B and let A and b as defined for (0). Furthermore, let 

X := { x G Q 2lJU ' I Ax > b, x > }. 
Then £ |= (A — > B,s,c) if and only if 

s < min{ 2_. x x \ x G X } 

xda , s 

(4) 



< min{ xx — c xx x G X }. 



XDAUB XDA 



G 



Proof Let C\= (A — > B,s,c). Then for every formal context K with attribute set M 
and K \= C it holds 

supp K (A — > B)>s, 
coniic(A — > B) > c, 

and so for the corresponding solution xk of ([3]) we have 

^2 x x > s, 

XDA 

^ X X - X X > C. 

XDAvB XDA 

Since by Theorem \§\ every solution of © originates from a model K of C, the inequal- 
ities (H} are satisfied. 

Conversely, let C ^= (A — > B. s, c). Then by LemmaUthere exists a formal context 
IK with attribute set M such that K j= C and K ^ (A — s- B, s, c), i. e. 

supp K (yl — > B) < s or 
confK(A — B) < c. 

Then the corresponding solution xr of Q satisfies 

^ < s or 

XDA 

XDAuB XDA 

and so the inequalities (H]) are not satisfied. □ 

4 Deciding Entailment in Polynomial Space 

The main disadvantage of the above approach is the size of the system ([3]) , which is 
exponential in \M\ and might therefore be exponential in the size of the original input 
£ and r = (A — > B.s.c). The resulting linear programs (jU) might therefore have 
a number of variables which is exponential in the number of inequalities. The case, 
however, that the number of variables is much larger than the number of restrictions 
is quite common in linear programming and algorithms have been devised to handle 
them specially. 

The purpose of this section is to represent such an algorithm. We shall show that 
this algorithm then is able to solve ([!]) in space polynomial in the size of the original 
input C and r. 
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To properly formulate this claim we need to agree on an encoding system for C 
and r, for otherwise the notion of ■polynomial space in the size of the input would be 
meaningless. See [2] for more details. 

We shall agree on the convention that integers are stored as binary numbers. Ra- 
tional numbers can be stored as pairs of integers. Matrices and vectors are stored 
as sequences of their entries, in row-major order. Finally sets are represented as se- 
quences of the elements they contain. For convenience we shall assume that we can use 
special characters like braces, parentheses and commas to separate different entities. 
This only increases the space requirements by a negligible amount. 

As the following description of the algorithm gets technical at certain points we 
may adopt some syntactical conventions to make this description more pragmatic. 

If x is a rational vector, we may write x = (xi,X2) to denote two vectors Xi,X2 
whose entries are a partition of the entries of x. That does not necessarily mean 
that the vector x must be ordered such that Xi comes first and X2 second. The 
same notation may be used to denote column partitions of matrices, i. e. that the 
columns of two matrices Ai, A2 constitute the columns of a matrix A may be written 
as A = (Ai, A2), without any special assumptions on the ordering of any columns. 
Finally, the columns of a matrix A are denoted by (ai)^!, n , if A has n columns. 

4.1 The Revised Simplex Method 

One of the best known algorithms for solving linear programs is the Simplex Method 
devised by George Dantzig. The revised Simplex Method is a variation of the original 
Simplex Method that reduces the space needed for the actual computations. The 
revised Simplex Method is mathematical folklore and we shall only give a short and 
compact representation of the method without giving any proofs. For further details, 
we refer to any introductory text on linear programming. The following description 
follows [1(3]. 

Let us consider the general linear optimization problem 

max{ c T x I x £ Q™, Ax = b, x > } (5) 

where A £ Q™x™ &m \ b g Q™. Moreover, we assume m < n and that A has maximal 
rank, i. e. rank(A) = m. Then the vertexes of the polytopeQ P = { x | Ax = b } 
contain optimal solutions for (O, and the Simplex Method (as well as the revised 
Simplex Method) goes along the edges of the polytope to find an optimal vertex. 
Vertexes of P correspond to maximal independent sets B = { , . . . , } of certain 
columns of A, and since A has full rank, k = m and the set B can be understood as 
a regular submatrix B of A. Let us denote with N the submatrix of A that contains 
all columns except those in B. 

Starting with such a submatrix B, which corresponds to a vertex x of P, the 
revised Simplex Method now tries to find another vertex x' such that c T x < c T x' . If 
such a vertex exists, then there also exists a vertex which is linked to x by an edge 

A polytope is finite intersection of halfspaces in Q™. 
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of the polytope P. To find such a vertex x', which amounts to find a set B' or the 
corresponding submatrix B' of A, the revised Simplex Method proceeds as follows: 

i. Subdivide the vectors x = (xb,xjv) and c = (cb,cjv) according to the chosen 
columns of A in B. Then xm = and xb = B x b. 

ii. Compute r = — c|^B" 1 N. 

iii. If all entries in r are non-positive, then x is an optimal solution for ([5]). 

iv. Otherwise let k be such that the fcth entry in r is negative. 

v. If all entries in the fcth column of B _1 N are non-positive, then © does not have 
a solution (the value of c T x is unbounded over P). Abort. 

vi. Otherwise let s be an index such that -j"- is minimal with d s k > 0. Here Xj is 
the jth component of and the entries of B _1 N are the elements d%j. Thereby 
choose s such that the corresponding column a s of A is the lexicographically 
smallest column such that -r 2 - is minimal. 

vii. Now the sth column a s of A is an element of B and the fcth column of A is 
not an element of B. Define B' := B \ { a s } U { a*, }. 

It can be shown that the solution x' corresponding to the set B' is a vertex of 
P such that c T x' > c T x. Iterating this procedure until it either returns an optimal 
solution or the information that no optimal solution exists constitutes the optimization 
procedure of the revised Simplex Algorithm. By (10.6) of 0], this algorithm always 
terminates. 

Now that we know the necessary details of the revised Simplex Method, we are 
able to prove the following theorem. 

Theorem 8 Let A e Q mxn ,b G Q"\c,x' G Q n such that m < n, rank(A) = m and 
Ax' = b. Let k be the largest amount of space needed to store any entry of A, b, c. 
Then the linear program 

max{ c T x | x G Q n , Ax = b,x>0} 

can be solved in space C(fcm 3 ) in addition to the space needed to represent the matrix 
A and the vectors b and c. 

Proof To see that the statement holds we have to show that we can execute all steps 
of the revised Simplex Method in not more that 0(km 3 ) space or that we can replace 
them by equivalent computations which can be done in space 0(fcm 3 ). 

We start by observing that we only need to store the indexes of the elements of 
B, xb and cb- This can be done in space O(km). Computing the inverse can be 
done in space C(fcm 3 ), using the adjugate matrix of B. This involves computations 
of determinants of B and of submatrices of B with one less row and column. Those 
computations can be carried out in time 0(km 3 ) each and hence do not need more 
space than 0(km 3 ). 
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The steps (|n |) - (jiv)) can be replaced by a simple loop that computes every component 
of r separately and returns the index of the first positive component. For this we note 
that we only need to compute c^B -1 once, yielding a row vector. After this, we 
successively compute scalar products with rows of N and save the index of the first 
row where the result is negative. If there is no such index, the algorithms stops. All 
this can be done in space 0{km). 

It is clear that we can execute step (jvj) in linear space, since the columns of B _1 N 
have m entries. 

Step ((vi]) iterates through all the indexes s of columns a s in B. For those indexes 
the value jp- needs to be checked for being negative, which can be done without 
actually computing the fraction. Moreover, comparing one such value with another 
can also be done without computing the value of the fraction. Finally, the column a s 
may have to be compared to another column vector for being lexicographically smaller. 
This can be done in space 0(km) and hence the overall step can be executed in space 
0{km). 

Finally, we note that updating the value for B requires only space logarithmic in 

m. 

So overall, one optimization step of the revised Simplex Method requires not more 
than 0(km 3 ) of space in addition to the space needed to store the original input. □ 

It may be noted that the revised Simplex Method (and the Simplex Method as 
well) consists of two phases, from which we only have described the second. Since we 
suppose a starting solution x, the description as it stands might not be very helpful in 
practice, since such a solution might not be given (and might not even exist). However, 
in our special circumstances we can explicitly give such a solution, as we shall see in 
the next subsection. 

4.2 Deciding Entailment in Polynomial Space 

With the revised Simplex Method at hand we are able to show that the linear pro- 
grams (UJ) can be solved in polynomial space. The main obstacle is to transform 
into the form of (0). 

Let us consider the system ([3]). We can convert it into the form of ([5]) by adding 
slack variables y.i for each inequality in (JTJ) . More precisely, we can transform the 
inequalities (P) into the equations 

^2 %A - Ui,i = Si, 

ADAi 

2J X A - C t 2J %A - Vi.2 = o, 
ADAiUBi ADAi 

such that yij > for all i G { 1, . . . , |/| },j G { 1, 2 }. We can do likewise for the 
inequalities in ^ by introducing y\ and ?/2- Hence we obtain the system 

Ax-y = b, x,y>0 (6) 
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where y = ((yi,j)igz,je{o,i}j2/i)2/2)' The following observation is well known in the 
theory of linear programming. 

Lemma 9 If x is a solution of J2P, then (x,y) £ (Q) 2 ' J| + 2 I 7 I+ 2 is a solution of (0) where 
y = Ax b > 0. Conversely, if (x, y) is a solution of (0), then x is a solution of J3]). 
In particular, the linear programs HP and the corresponding programs 

min{ a?x | x e A" }, 
min{ x x — c x x x £ ^} 

XDAuB XDA 

with 

# = {x| Ax y = b,x,y>0} 

have the same values, respectively. 

We are now ready to formulate the overall goal of our considerations. 

Theorem 10 The systems 0) can be solved in polynomial space. 

Corollary 11 The linear programs HP can be solved in polynomial space. Therefore, 
entailment of constrained implications can be decided in polynomial space. 

Proof (Theorem\l(J\) Let m = Let us write the system ((SJ) as 

Ax = b, x > (8) 

Where A = (A, -I) G Q(2m+2)x(2 m +2r„+2) and - = ( X;y ) g Q2'"+2m+2^ jj^ J 

denotes the identity matrix of the corresponding size. Note that A has full rank, since 
it contains —I as submatrix. 

We start by noting the following two facts, which we shall discuss in detail after- 
wards: 

i. The matrix A can be represented implicitly, i. e. the entries of A can directly be 
inferred from the input C and r. 

ii. A solution for |SJ can be given explicitly. This is due to the fact that we can hnd 
an explicit solution for ((3]), namely xm = 1 and xa = for A C M. 

We start with point ©. For this let i £ { 1, . . . , m + 2 }, j £ { 1, . . . , 2 m + 2m + 2 } 
and A = (a s t) s .t- Then we can distinguish the following cases: 

1. If j > 2 m , then if i = j — 2 m , then 5y = 1, otherwise 6»j = 0. 

2. i = 2m + 1 and j < 2 m , then ckj = 1. 

3. i = 2m + 2 and j < 2 m , then a i3 = -1. 
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4. i = 2k for some k E I, j < 2 m . Let C be the set corresponding to the index j. 
Then = 1 if C 2 ^4fc and a^- = otherwise. 

5. i = 2k + 1 for some fc s /, j < 2 m . Again let C be the set corresponding to the 
index j. If C D A k U B k , then C D A k and a„ = 1 - c fc . If C 2 ^fc U but 
C 2 ^4fc, then fiy = — c^. Otherwise, 5y = 0. 

The last two cases can be seen from the inequalities ([I}. Hence, A can be inferred 
from C and r and need not be stored explicitly. 

For the point ^ let x as discussed there. Then let y = Ax — b. Then B = 
{a2m,a2m +1 , . . . ,a2m+2m+i } is a set of linearly independent columns of A that cor- 
respond to the solution (x, y) . 

Now we can apply Theorem [5] to see that we only need 0(km 3 ) additional space 
to solve (O, where k is the largest amount needed to store an entry of A, b, c. This 
however reduces to the largest amount needed to store any value of Sj or a, which are 
part of the input. Since we have not used more than C(fcm 3 ) space so far, we can 
hence solve ((5J in polynomial space in the size of the input and the claim is proven. □ 

5 Conclusions and Further Research 

Starting from the motivation to understand association rules as logical objects usable 
for reasoning, we have introduced the notion of constrained implications. Based on 
this definition we have developed semantics for constrained implications based on 
models and stated the corresponding entailment problem. We then investigated a 
reformulation of instances of the entailment problem as pairs of linear programs and 
were able to show that those linear programs can be solved in polynomial space. 

The result that entailment for constrained implications can be decided in poly- 
nomial space does not seem that pleasing. Above all, it does not give reasonable 
constrains on the time needed to decide entailment, a fact that might be of practical 
interest. 

A better complexity bound might hence be desirable. It is quite unclear whether 
one can expect the entailment problem to be an element of either AfV, co-NT or 
even V . It might, however, be worth looking into complexity classes of the polynomial 
hierarchy whether the entailment problem is located there. 

Another problem, which is closely connected to our original motivation but which 
has not been addressed in this work is the following: Since we have established a notion 
of entailment for constrained implications it might be worth searching for minimal 
non-redundant sets of constrained implications representing conImp s C (K) for a formal 
context K and s,c£ [0, 1] HQ. Results in this direction might give more insight in the 
usefulness of regarding association rules as logical objects. 
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